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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Patent Application of 

MILNER Atty. Ref.: 36-1470 

Serial No. Unknown Group: 

National Phase of: PCT/G BOO/01 052 
International Filing Date: 21 March 2000 

Filed; August 24, 2001 Examiner: 

For: HANDWRITING RECOGNITION SYSTEM 

August 24, 2001 

Assistant Commissioner for Patents 
Washington, DC 20231 

Sir: 

PRELIMINARY AMENDMENT 

Prior to calculation of the filing fee and in order to place the above identified 
application in better condition for examination, please amend the claims as follows: 
IN THE CLAIMS 

Please substitute the following amended claims for corresponding claims 
previously presented. A copy of the amended claims showing current revisions is 
attached. 

3. (Amended) A handwriting recognition system as claimed in claim 1 , in which 
the sampling means, filtering means and classifier are implemented in a digital 
computer environment. 



MILNER 

Serial No. Unknown 

REMARKS 

Attached hereto is a marked-up version of the changes made to the claims by the 
current amendment. The attached page is captioned " Version with markings to show 
changes made . " 

The above amendments are made to place the claims in a more traditional 
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11 00 North Glebe Road, 8th Floor 
Arlington, VA 22201-4714 
Telephone: (703)816-4000 
Facsimile: (703) 816-4100 



format. 



Respectfully submitted, 
NIXON & VANDERHYE P.C. 
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MILNER 

Serial No. Unknown 

VERSION WITH MARKINGS TO SHOW CHANGES MADE 

3. (Amended) A handwriting recognition system as claimed in claim 1 [or claim 
2], in which the sampling means, filtering means and classifier are implemented in a 
digital computer environment. 
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HANDWRITING RECOGNITION SYSTEM 



The present invention relates to a handwriting recognition system and more 
particularly to a hardware and an algorithm for implementing such a system. 



is disclosed a portable computer in the form of a pen-type casing. Incorporated 
within the casing is at least one accelerometer which is used to detect movement of 
the pen with respect to its environment. By using the instrument for handwriting it is 
possible to effect data entry or transmission of signals reflecting movement, the user 

10 using either a pen tip mounted switch or a finger operated switch to indicate that 
movement is effecting a written input. 

There are many other pen-type input devices on the market in addition to 
stylus scroll pallets typically used in so-called paim top computers where handwriting 
recognition has been used. Such devices often require very precise movement which 

15 may not reflect natural handwriting movements for the user. One of the problems 
which makes characteristic handwriting recognition difficult is that while the 
underlying movement made by an individual to represent a particular letter may be 
consistent an element reflecting user movements due to stress and other factors will 
be present. 

20 According to the present invention there is provided a handwriting 

recognition system comprising means responsive to input analogue signals 
representative of movement of a handheld writing device, sampling means to provide 
signals representative of the acceleration of the writing device in at least x axis and y 
axis channels at a predetermined capture sampling rate, and filtering means to 

25 remove do level components and to provide smoothing of the output whereby signals 
representative of movement of the pen over a period are supplied to a classifier for 
comparison with a template representative of characters formed. 

The classifier may use hidden Markov modelling (HMM) techniques using a 
large number of states to determine the character defined by movement. The system 

30 may include an input indicative of a user's intention that the movement is 
representative of character writing. 

According to a feature of the present there is provided a method of analysing 
signals from a moving handheld device, the method comprising sampling signals at a 
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predetermined rate, passing signals through a bandpass filter to remove dc level and 
excess acceleration components, sampling the filtered output to provide a series of 
vectors representing the position of the handheld device at periodic intervals and 
using a classifier to compare the sample sets with predetermined templates to 
5 determine the character for output. 

A handwriting recognition system in according with the invention will now be 
described by way of example only with reference to the accompanying drawing of 
which: 

Figure 1 is a block schematic diagram of the system; 
10 Figure 2 shows relative positioning of the x and y axis of the handwriting 

device of Figure 1 ; 

Figure 3 is a schematic diagram of the handheld writing device of Figure 1 in 
a particular position; 

Figure 4 shows relative input and output vector streams from the system of 
1 5 Figure 1 ; and 

Figures 5 to 9 show comparative templates for a number of different letters. 

Referring first to Figure 1, an input device 1 such as a stylus produces x and 
y vector streams 2 and 3 which are fed into a sampling unit 4. The outputs x and y 
are generally from accelerometers or other position sensing devices within the stylus 
20 1. Also feeding the sampling unit 4 is an output 5 from a switch indicated here as 
being in the nib section of the stylus 1 such that contact between the switch 6 and a 
surface is indicative of the stylus being used in a writing mode. It will be appreciated 
that the nib switch 6 when incorporated in a non-surface contacting stylus, such as 
that disclosed in the previously referred PCT application, may be replaced by a user 
25 operable switch. 

The output of the sampling unit 4, which samples the incoming streams at 
60 Hz for example, is passed to a bandpass filter arrangement 7 and thence to a 
down sampling unit 8 which produces digitised vectors x and y over a period of time. 
The X and y vectors are passed to a classifier 9, which uses a hidden Markov model 
30 to carry out a comparison between the vectors and templates representing written 
characters. The classifier 9 may be arranged to output to a visual display 10. 

More specifically, the stylus 1 for example comprises a simple plastic casing 
containing the electronics for transferring information to a PC. Two accelerometers 
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mounted in the stylus, for example, are used to produced. the x and y outputs. The 
nib switch is a simple on/off switch connected to determine when pressure is being 
applied to the pen nib and can therefore detect when a pen for stylus 1 is writing. 

Turning briefly to Figure 2, the two accelerometers mounted in the top of the 
5 pen measure acceleration across their plane such that effectively they measure 
acceleration along the x axis 11 and the y axis 12 of the writing surface 14. The 
acceleration measured by the sensors is made up of two components, acceleration 
due to gravity and the acceleration as a result of stylus movement. It will be 
appreciated that the acceleration due to gravity is always present, such that when 
10 the pen is exactly horizontal both sensors would measure acceleration of ^g. As the 
angle of the pen to the horizontal changes (as shown for example in Figure 3), the 
accelerometers are subject to Sin Qx^g where 6 is the inclination angle of the stylus 
1. 

The other component, acceleration as a result of the pen moving is produced 
15 by the acceleration and deceleration effect as the user writes. 

The acceleration of the two sensors Xtotai and y^otai can be expressed as 

ytotal = Vg + ymovement a^ld X„tal = + X^^^^^^nt 

20 The remaining items of Figure 1 are incorporated in a computer unit, for example a 
PC, and three signals, as previously indicated, 2, 3 and 5 being the two acceleration 
signals and binary signal from the pen switch are provided to the PC. 

in one embodiment the two acceleration signals are read into a normal PC 
using an RS232 port and the binary switch signal by means of the games port of a 

25 sound card. 

The sampling section must sample sufficiently regularly to capture the 
movement of the pen but should not over-sample, which would result in a waste of 
processing and storage within the PC. It has been found satisfactory for the 
purposes of the current invention to sample at a rate of 60 Hz. The acceleration 
30 signals for each channel are read in as two byte words giving a dynamic range for 
each acceleration signal from 0 to 65535. The pen nib switch is similarly sampled at 
the same rate. 
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As previously mentioned, the accelerometer signals are partly dependent on a 
component of the earth's gravitation field passing through the accelerometer of the 
stylus 1. This results in an almost constant dc level present on the output 
corresponding to the average pen angle 9 while writing. The bandpass filter 7 is thus 
5 arranged to filter the signals from the two accelerometers to remove the offsets. 
Additionally, the bandpass filter smoothes the output from the sensors thus 
correcting for instability introduced by the user so that the smoothed output from the 
sensors increases robustness and facilitates matching between the x and y vectors 
and stored templates. 

10 Turning now to Figure 4, for each sample received on the PC from the 

acceierometers, the pen nib switch (or manually operable switch) indicates whether 
the stylus 1 was being used in a writing mode or not. The down sampling process 8 
uses the information to down sample acceleration samples and to retain only those 
when it was known that the stylus 1 was writing. Thus, consider Figure 4, assuming 
15 that the x axis and y axis vectors are as shown at 15 and 16, then the down 
sampling vectors need only be taken into account when the nib switch signal 
indicator 17 is high. This will reduce the number of samples significantly so that the 
output from the down sampling process is a time series of two dimensional vectors x 
and y as indicated at 18 and 19. 
20 Having completed processing of the acceleration measurement from the 

stylus 1 the vector stream is passed to the classifier stage 9 which takes in a series 
of vectors representing the acceleration measurements made within a given word. 
These are then compared to a set of templates which cover the range of words within 
the system vocabulary and the word which matches most closely with the unknown 
25 input word is deemed the recognised word. In this system the classifier is a hidden 
Markov model. Such models have been widely used in speech recognition and using 
a large number of states in the hidden Markov model will give the best performance 
for corresponding handwriting recognition. 

The display 10, which displays the output from the PC allows display of a 
30 word, for example, on a screen. 

While the above handwriting recognition system is intended for use with a 
series of known words which, depending on the system vocabulary entered into the 
PC, may be a large number, it will be possible to use the same kind of system to 
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validate single character entry. Using single character recognition and using cursive 
entry it is still possible to build individual words which may not be present in the 
vocabulary. There may be a lower level of confidence in words created rather than 
template determined. However, over time, the vocabulary may be expanded where 
5 multiple entries of the same word have occurred such that higher confidence levels 
may be achieved. 

A typical single accelerometer output can be seen respectively for the letters 
c, b, f and h in Figures 5 to 8. In each case the template developed here shows three 
entries on a single accelerometer for each of the letters. 

10 In Figure 9, f, b, h and c are shown in comparison so that a suitable template 

for comparison may be derived. It will be appreciated that the combination of an x 
accelerometer trace and a y accelerometer trace will serve further to emphasise the 
difference between each input letter. 

Bandpass filtering in digital form to remove dc components and high 

15 frequency components increases the reliability of the recognition process and 
therefore the reliability of the interpretation of the stylus input 1. It will be 
appreciated that where the stylus 1 carries other components, for example an internal 
processing arrangement, some of the functions may be transferred from the PC to the 
stylus 1. All of the components of sampling, bandpass filtering, down sampling and 

20 classifying can be implemented in a suitable computer program. 
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CLAIMS 

1 . A handwriting recognition system comprising: 

means responsive to input analogue signals representative of movement of a 
handheld writing device; 

sampling means to provide signals representative of the acceleration of the 
writing device in at least x axis and y axis channels at a predetermined capture 
sampling rate; and 

filtering means to remove dc level components and to provide smoothing of 
the output whereby signals representative of movement of the pen over a period are 
supplied to a ciassifier for comparison with a template representative of characters 
formed. 

2. A handwriting recognition system as claimed in claim 1, in which the 
classifier uses a hidden Markov model for comparison purposes. 

3. A handwriting recognition system as claimed in claim 1 or claim 2, in which 
the sampling means, filtering means and classifier are implemented in a digital 
computer environment. 

4. A method of analysing signals from a moving handheld device, the method 
comprising sampling signals at a predetermined rate, passing signals through a 
bandpass filter to remove dc level and excess acceleration components, sampling the 
filtered output to provide a series of vectors representing the position of the handheld 
device at periodic intervals and using a classifier to compare the sample sets with 
predetermined templates to determine the character for output. 




ABSTRACT 
HAMDWRITING RECOGNITION SYSTEM 

In order to improve the accuracy of recognition of hand-written input using a 
5 stylus (1), output signals from a plurality of accelerometers representing x and y axis 
acceleration and deceleration are sampled at a predetermined rate and passed through 
a digital bandpass filter (7) to remove high frequency components and dc components 
arising from gravity, x and y vectors derived from the original x and y input signals 
are passed to a classifier using a hidden Markov model. Bandpass filtering improves 
10 the robustness of the interpretation of the vectors against stored templates which 
may be templates of individual characterisations or of whole words. 

Figure 1 
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RULE 63 (37 C.F.R.1.63) 
DECLARATION AND POWER OF ATTORNEY 
FOR PATENT APPLICATION 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

As a below named inventor, I hereby declare that my residence, post office address and citizenship are as stated below next to my name and I believe 
I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are listed below) of the 
subject matter which is claimed and for which a patent is sought on the Invention entitled: 

HANDWRITING RECOGNITION SYSTEM 



the specification of which (check applicable box(s)): 

□ Is attached hereto „, , ^, 

□ was filed on as U.S. Application Serial No. (Atty DM. No. 

^ was filed as POT International application No. POT/GB 00/01052 on 21 March 20Q0 

and (if applicable to U.S. or POT application) was amended on . _ 

1 hereby state that I have reviewed and understand the contents of the above identified specification, Including the claims, as amended by any 
amendment referred to above. I acknowledge the duty to disclose information which Is material to the patentability of this application in accordance with 
37 C F R 1 56 I hereby claim foreign priority benefits under 35 U.S.C. 1 1 9/365 of any foreign application(s) for patent or inventors certificate listed 
below and have also Identified below any foreign application for patent or inventor's certificate having a filing date before that of the application on which 
priority is claimed or, if no priority is claimed, before the filing date of this application: 

Priority Foreign Application(s): Dflv/Month/Year Filed 



I hdriiy claim the benefit under 35 U.S.C. §11 9(e) of any United States provisional application(s) listed below. 
Application Number Date/Month/Year Filed 

I hliybv claim the benefit under 35 U.S.C. 120/365 of all prior United States and PCX international applications listed above or below and, insofar as the 
suWm matter of each of the claims of this application is not disclosed in such prior applications in the manner provided by the first paragraph of 35 
uMq. 1 12, 1 acknowledge the duty to disclose material information as defined in 37 C.F.R. 1.56 which occurred between the filing date of the pnor 
ap^fetions and the national or PCX international filing date of this application: 

Pri^mU.S7PCT Application(s): Status: patented 

" ^ - ^ - ^ Day/Month/Year Filed pending, abandorted 



Appfication Serial No. 



piF/GBOO/01052 21 March 2000 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and belief are believed to 
be We- and further that these statements were made with the knowledge that willful false statements and the like so made are punishable by fine or 
imirfeonment or both under Section 1001 of Xitle 18 of the United States Code and that such willful false statements may jeopardize the validity of the 
. ' ... _x.i /-\ 1,^, — f I h^>^hw or.r.r,.r.t Mivr>M s. vAMnPRHVF PC 1 1 QQ Nopth G lebo Rd.. 



application or any patent issued thereon. And on behalf of the owner(s) hereof, I hereby appointJvHXQIvL&JiANDIRtDCEP.C., . 
8^ ]_Floor. Arlington. VA 22201-4714 , telephone number (703) 816-4000 (to whom all communications are to be directed), 



and the following 



— , ... the Patent and Trademark Office connected therewith and with the resulting patent: Arthur R. Crawford, 25327; Larry S. Nixon, 25640; Robert A 
ni Vand6rhve,27QZ6; James T. Hosmer, 30184; Robert W. Paris, 313^ Richard G. Besha.^TTO; Mark E: Nusbauni^3234,8; MichaeTD. Keenan, 32106;_ 
^ Bryan H Davidson, 30251; Stanley C. Si^er,JZ3a3; Leonard C. Mitchard, 29009; Duane M. Byers, 33363; Jeffry H. Nelson, 30481; John R. 
Lastova, 33149; H. wi^Ten Bumam, Jr. 29366; Thomas E. Byrne, 32205; Mary J. Wilson.,32^5; J. Scott DavidsoaJ334a9; Alan M. Kagen, 
Robert A Molan, 29834; B. J. Sadoff, SgmTJames D. Berquist, 34776; Updeep S. Gill, 37334iMichael J. Shea,_34725;J3onald L. Jackson, 4jum- 
Michelle N LesterT^SSI ; Frank P. Presta, 1JS2auloseph S. Presta, 35329 1 also authorize Nixon & Vanderhye to delete any attorney 
names/numbers no longer with the firm and to act and rely solely on IrlstFucIons directly communicated from the person, assignee, attorney, firm, or 
other organization sending instructions to Nixon & Vanderhye on behalf of the owner{s). 
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